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Abstract 

This  paper  presents  a  success  story  of  specifying  a  complex  real-life  protocol  (MIL-STD  188-220)  in  Estelle  and  generating  test  sequences 
from  the  formal  specification.  188-220  is  being  developed  in  the  US  Army,  Navy  and  Marine  Cotps  systems  for  mobile  combat  network 
radios.  A  key  factor  in  this  success  story  has  been  the  collaboration  among  the  researchers  of  the  University  of  Delaware  and  the  City  College 
of  the  City  University  of  New  York,  the  developers  of  the  US  Army  Communications-Electronics  Command  (CECOM),  and  the  protocol 
designers  in  the  Joint  Combat  Net  Radio  Working  Group.  Based  on  the  research  results,  188-220  test  sequences  are  realizable  without  timer 
interruptions  while  providing  a  200%  increase  in  test  coverage.  The  test  cases  are  being  installed  at  a  CECOM  test  facility.  ©  2000  Elsevier 
Science  B.V.  All  rights  reserved. 
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1.  Introduction 

Formal  methods  in  conformance  testing  are  becoming 
widely  used  in  the  testing  of  real-life  protocols 
[6,16,17,24,35,36,76].  OSI  conformance  testing  has  been 
applied  to  the  Internet  protocols  such  as  Flypertext  Transfer 
Protocol  [24],  Simple  Mail  Transfer  Protocol  [6],  and  TCP/ 
IP  [36].  Another  area  of  successful  application  of  confor¬ 
mance  testing  are  ATM/B-ISDN  protocols,  e.g.  the  testing 
of  ATM  Adaptation  Layer  protocol  [76],  ATM  switching 
systems  [16,35],  and  B-ISDN  signalling  [17]. 

There  have  also  been  several  reports  on  successful  appli¬ 
cation  of  Estelle  to  real-life  systems  [8,14,34,49,58,69], 
Estelle  was  used  to  automatically  implement  TP4/IP  proto¬ 
col  [69],  to  uncover  ISO  Remote  Operations  Service 
Element  [29]  protocol  errors  [34],  and  to  specify,  detect, 
and  resolve  feature  interactions  in  Intelligent  Networks 
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[8],  Most  recently,  simulation  studies  [49]  of  the  Estelle 
specification  of  Service  Specific  Connection-Oriented 
Protocol  [33]  revealed  some  specification  errors  and  flow 
control  inefficiencies  of  the  protocol  definition.  Catrina  et  al. 
[14]  showed  that  it  is  possible  to  use  an  Estelle  specification 
to  automate  implementation  of  a  sophisticated  transport 
protocol  (XTP  4.0  [75]).  Thees  [58]  compared  performance 
of  the  XTP  4.0  implementations  automatically  generated 
from  the  Estelle  specification  with  hand-coded  implemen¬ 
tations.  Both  studies  point  out  numerous  advantages  (and 
potential  problems)  of  using  Formal  Description  Tech¬ 
niques  (FDTs)  for  XTP  code  generation. 

This  paper  presents  a  success  story  of  specifying  a 
complex  real-life  protocol  in  Estelle,  and  automatically 
generating  conformance  tests  from  the  formal  specification. 
The  US  Department  of  Defense  (DoD)/Joint  protocol,  called 
Military  Standard  (MIL-STD)  188-220,  is  being  developed 
in  the  US  Army,  Navy  and  Marine  Corps  systems  for  mobile 
combat  network  radios  [19].  With  the  understanding  of  the 
power  of  formal  description  techniques,  a  key  factor  in  this 
success  story  has  been  the  collaboration  among  the  four 
groups:  the  researchers  of  the  University  of  Delaware 
(UD)  and  the  City  College  of  the  City  University  of  New 
York  (CCNY),  the  developers  of  the  US  Army 
Communications-Electronics  Command  (CECOM)  in 
New  Jersey,  and  the  protocol  designers  in  the  Joint  Combat 
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Fig.  1.  History  of  MIL-STD  188-220  development. 


Net  Radio  (CNR)  Working  Group  (WG).  As  a  result  of  this 
success  story,  the  synergistic  framework  to  develop  C4/ 
(Command,  Control,  Communications,  Computers,  and 
Intelligence)  systems  with  the  help  of  formal  methods  serves 
as  a  model  for  future  DoD  standards  development  [19]. 

Since  this  paper  is  a  case  study  promoting  a  successful 
application  of  Estelle  to  a  real-life  protocol,  it  includes  a 
cross-section  of  activities  over  the  past  few  years.  Section  2 
provides  the  background  on  the  collaboration  among  the 
MIL-STD  188-220  sponsors,  research  and  development 
teams,  and  standards  organizations.  Sections  3  and  4  briefly 
overview  the  formal  description  technique  Estelle  and  188- 
220,  respectively.  Section  5  presents  a  part  of  the  Estelle 
specification  of  188-220,  and  gives  examples  of  errors  and 
ambiguities  found  thanks  to  formally  specifying  the  proto¬ 
col.  A  general  approach  adopted  at  UD  and  CCNY  to  test 
generation  from  an  Estelle  formal  specification  is  described 
in  Section  6.  This  section  also  summarizes  recent  research 
results  obtained  in  minimum-length  test  generation  based  on 
Estelle  specifications.  Section  7  contains  information  on 
practical  results  in  test  generation  and  delivery.  Finally, 
Section  8  presents  the  authors’  view  on  the  improvements 
to  the  protocol  development  process  due  to  using  formal 
methods. 


2.  History  of  MIL-STD  188-220  development 

In  1994,  UD’s  Protocol  Engineering  Laboratory  began  its 
involvement  with  the  US  Army  in  using  Estelle 
[11,28,57,60]  to  formally  specify  a  military  standard  MIL- 
STD  188-220  [18].  An  initial  small  contract  with  the  Army 
Research  Laboratory  supported  both  simulation  and  speci¬ 
fication  of  the  1993  version  of  188-220  [13,45].  The  formal 
specification  research  effort  received  the  attention  of  the 
CECOM  Software  Engineering  Center,  which  leads  the 
effort  to  evolve  188-220  to  meet  the  Army’s  requirements 
for  battlefield  digitization,  through  the  Joint  CNR  WG,  itself 
responsible  for  the  evolving  188-220  standard. 

From  1995  to  1998,  at  least  fifty  changes  to  the  English 
specification  of  188-220  resulted  from  UD’s  efforts  using 
Estelle  to  formally  specify  the  standard  [2,18]  (see  Fig.  1). 
While  the  English  text  takes  precedence  in  case  of  disagree¬ 
ment  with  the  formal  specification,  the  Estelle  specification 


of  188-220  is  an  official  part  of  the  military  standard.  To  the 
best  of  our  knowledge,  1 88-220 1  represents  one  of  the  first 
major  national  or  international  standards  officially  including 
an  Estelle  specification.  There  are  other  examples  of  inter¬ 
national  standards  that  include  Estelle  specifications  such  as 
OSI  session  protocols  [30,48],  OSI  Distributed  Transaction 
Processing  [31],  and  OSI  File  Transfer,  Access  and  Manage¬ 
ment  protocol  [27] . 

CECOM  is  developing  a  Conformance  Tester  that  auto¬ 
matically  evaluates  a  188-220  implementation  identifying 
where  it  differs  from  the  standard.  The  test  generation 
research  was  initiated  as  part  of  the  US  Army’s  Advanced 
Telecommunication  and  Information  Distribution  Research 
Program  (ATIRP)  in  January  1996,  when  UD’s  Protocol 
Engineering  Lab  began  research  collaboration  with 
CCNY.  Efforts  were  focused  on  automatically  generating 
test  cases  from  the  Estelle  specifications.  Generating  tests 
from  formal  specifications  such  as  pure  finite  state  machines 
(FSMs)  has  been  extensively  studied  in  the  literature.  But 
the  inherent  complexity  of  188-220  is  far  beyond  specifying 
with  pure  FSMs,  hence  the  need  to  use  a  more  powerful 
specification  language  such  as  Estelle,  ISO  9074.  Unfortu¬ 
nately,  generating  tests  from  Estelle  specifications  presents 
difficult  theoretical  and  practical  problems.  UD  and  CCNY 
faculty  and  students  continue  to  investigate  these  problems 
with  the  practical  motivation  of  applying  the  results  towards 
188-220  test  case  generation. 

As  mentioned  above,  automatic  generation  of  tests  from 
Estelle  specifications  presented  various  theoretical 
problems: 

1.  During  testing,  if  active  timers  are  not  taken  into  account, 
they  can  disrupt  the  test  sequences,  failing  correct  imple¬ 
mentations  (or  worse,  passing  incorrect  ones).  Therefore, 
timers  have  to  be  incorporated  as  constraints  into  the 
extended  FSM  (EFSM)  model  of  an  Estelle  specification. 

2.  Test  sequence  generation  is  limited  by  the  controllability 
of  an  Implementation  Under  Test  (IUT)  [7].  Testers  may 
not  have  direct  access  to  all  interface(s)  in  which  the  IUT 
accepts  inputs  (typically,  the  interfaces  with  upper  layers, 
or  with  timers).  In  this  case,  some  inputs  cannot  be 
directly  applied;  the  interactions  involving  such  inter¬ 
faces  may  render  some  portions  of  the  protocol  untest- 
able,  and  may  introduce  non-determinism  and/or  race 
conditions  during  testing. 

3.  Infeasible  test  sequences  may  be  generated  unless 
conflicting  conditions  based  on  protocol’s  timers  that 
may  be  running  simultaneously  are  resolved  (the  so- 
called  conflicting  timers  problem). 

The  timing  and  controllability  issues  were  present  in  the 
EFSM  model  of  the  Estelle  specification  of  MIL-STD 
188-220  [3,20].  Based  on  the  results  of  the  investigating 
problems  (1)  and  (2)  by  the  UD  and  CCNY  joint  group 
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•  Behavior:  Communicating  Extended  FSMs  +  Pascal 

-  variables 

-  priorities 

-  delays 

-  conditionals 

•  Architecture:  Hierarchy  and  Interconnections  of  EFSMs 

-  modules 

-  interaction  points 

-  channels 


Fig.  2.  Estelle:  ISO  international  standard  9074. 


[20,23,66],  UD  has  been  providing  CECOM  with  auto¬ 
matically  generated  test  sets  since  1997.  The  sizes  of  the 
resulting  FSMs  derived  from  the  Estelle  specification  range 
from  48  to  303  states,  and  from  119  to  925  transitions.  The 
corresponding  test  sequences  range  from  145  to  2803  test 
steps.  These  tests  are  free  of  interruptions  due  to  unexpected 
timeouts  while  their  coverage  of  the  number  of  testable 
transitions  increased  from  approximately  200  to  over  700 
by  utilizing  multiple  interfaces  without  controllability 
conflicts.  Current  research  focuses  on  the  conflicting  timers 
problem. 


in  the  original  English  document  that  would  cause  interpre¬ 
tation  problems  for  implementors. 

An  Estelle  specification  consists  of  two  parts:  architecture 
and  its  behavior.  The  architecture  specifies  a  collection  of 
systems  of  nested  modules.  Each  module’s  behavior  is 
described  by  an  extended  FSM.  These  EFSMs  interact  via 
the  sending  of  interactions  over  a  set  of  channels.  The  inter¬ 
actions  are  conceptually  stored  in  infinite  FIFO  queues 
enabling  transitions  in  the  receiving  module  which  are 
fired  when  all  enabling  conditions  are  satisfied.  A  complex 
set  of  rules  define  either  a  parallel  or  synchronous  firing  of 
transitions  within  each  EFSM.  Overall,  the  many  features  of 
Estelle  allow  a  user  to  formally  specify  a  wide  variety  of 
network  protocol  behaviors.  Further  information  about 
Estelle  can  be  found  in  Refs.  [57,60]. 

One  major  benefit  of  an  Estelle  specification  as  a  model 
of  a  communication  protocol  is  that  it  can  be  used  as  input  to 
a  conformance  test  generation  tool.  Since  Estelle  makes  it 
possible  to  create  a  complete  and  unambiguous  protocol 
model,  the  test  cases  generated  from  it  can  potentially 
achieve  higher  fault  coverage  than  hand- generated  ones, 
and  are  reproducible  with  far  less  effort  as  188-220  evolves 
in  the  future.  These  advantages  are  the  primary  motivations 
for  using  Estelle  to  specify  188-220. 


3.  Estelle 


4.  MIL-STD  188-220 


In  1989,  Estelle  was  published  as  one  of  two  ISO  Inter¬ 
national  Standard  Formal  Description  Techniques  (FDT)  for 
the  specification  of  computer  communication  protocols 
[11,28].  As  shown  in  Fig.  2,  Estelle  specifies  a  protocol’s 
behavior  as  a  set  of  communicating  extended  finite  state 
machines.  To  avoid  ambiguity  among  different  readers  of 
a  specification,  the  Estelle  language  itself  has  a  formal, 
mathematical,  implementation-independent  semantics. 

Estelle  is  an  expressive,  well-defined,  well-structured 
language  that  is  capable  of  specifying  distributed,  con¬ 
current  information  processing  systems  in  a  complete, 
consistent,  concise,  and  unambiguous  manner.  An  Estelle 
specification  aims  at  discovering  and  resolving  ambiguities 


Fig.  3.  MIL-STD  188-220  protocol  architecture.  Parts  of  the  protocol  where 
FDTs  were  used  during  the  development  are  circled. 


The  Protocol  Engineering  Lab  researchers  at  UD  used 
Estelle  to  specify  parts  of  the  188-220  protocol  suite.  This 
suite  was  developed  to  meet  the  requirements  for  horizontal 
integration,  seamless  Internet  communications  and 
increased  mobility  using  combat  network  radios  [19].  This 
protocol,  a  critical  piece  of  the  new  Joint  Technical  Archi¬ 
tecture,  is  now  mandated  for  CNR  communications.  It  is 
being  implemented  in  US  Army,  Navy  and  Marine  Corps 
systems,  and  has  been  demonstrated  initially  during  the 
Army’s  Advanced  Warfighting  Experiment  in  1997.  188- 
220  is  now  receiving  allied/international  attention,  while 
portions  of  its  protocol  architecture  have  been  promulgated 
in  the  Internet  Engineering  Task  Force.  Expected  outcomes 
from  its  use  are:  seamless  connectivity  of  C4/  systems 
(discussed  briefly  in  Section  8),  horizontally  integrated 
information  networks,  and  joint  interoperable  C4/  systems 
for  the  warfighter. 

188-220,  originally  developed  in  1993,  evolved  to  188- 
220A  with  substantial  new  functionality,  including  support 
for  new  radio  technology  and  integration  with  Internet 
protocols  (commercial  IP,  TCP,  and  UDP  at  the  network 
and  transport  layers).  Version  188-220B,  whose  architecture 
is  depicted  in  Fig.  3,  describes  the  protocols  needed  to 
exchange  messages  using  CNR  as  the  transmission  media. 
These  protocols  include  the  physical,  data  link  and  part 
of  the  network  layer  of  the  OSI  model.  The  protocols 
apply  to  the  interface  between  host  systems  and  radio 
systems.  Hosts  usually  include  communications  processors 


M.A.  Fecko  et  al.  /  Computer  Communications  23  (2000)  1196-1213 


1199 


Network  Layer  Interface 


Network  Laver  -  IF,  SNDCF,  Intranet  Interface 


Transport  Layer 


7p,9,10,ll,i2f 


2,3 


Network  Laver 


13,14,15 


5,6 


Datalink  Layer 


1. NL-UaitdataRq 

2.  NL-Umtdata.]nd 

3. NL-Status.Lnd 

4. DL-Uiiitdata.Req  (DL-Umtdata-id*  1~16  dest  addr, 
src  addr,  top-id,  precedence,  throughput,  delay, 
reliability,  data,  data  length) 

5.  DL-Unitdata.lnd  (1-16  dest  addr,  src  addr,  top-id, 
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15.  OP-leave-netlnd 


7.  OP-join-net.Req 

8.  OP-leave-netReq 
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23.SNDCF-Status.Ind 

24.  TL-Unitdata.Req  (IL-Unitdata-id*,  message 
type,M6  dest  addr,  src  addr,  precedence, 
throughput,  delay,  reliability,  data,  data  length) 

25. 1L-Unitdata.lnd  fIL-Unitdata-id*  ,1-16  dest 
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26.  L-StatusJnd  (IL-Unitdata-id*,  ackfailure, 
intranet  path  status,  1-16  dest  addr) 


Fig.  4.  Network  layer  interface  and  architecture. 


or  modems  that  implement  these  lower  layer  protocols.  The 
unshaded  portions  of  Fig.  3  indicate  those  protocols  and 
extensions  that  were  developed  specifically  for  use  with 
CNR. 

MIL-STD  188-220  Datalink  layer  specifies  several 
service  types,  each  intended  to  handle  different  types  of 
traffic  with  different  quality  of  service  (QoS)  demands.  A 
188-220  station  can  actually  process  several  different  types 
of  traffic  simultaneously  (and  almost  orthogonally).  MIL- 
STD  188-220  Network  Layer  consists  of  Internet  (IP)  Layer, 
Subnetwork  Dependent  Convergence  Function  (SNDCF), 
and  Intranet  Layer.  The  Intranet  Layer  has  been  dedicated 
to  routing  intranet  packets  between  a  source  and  possibly 
multiple  destinations  within  the  same  radio  network.  The 
Intranet  Layer  also  accommodates  the  rapid  exchange  of 
topology  and  connectivity  information — each  node  on  the 
radio  network  needs  to  determine  which  nodes  are  on  the 
network  and  how  many  hops  away  they  are  currently 
located. 


5.  188-220  Estelle  specification 

To  help  a  reader  realize  the  magnitude  of  formally  speci¬ 
fying  a  protocol  of  188-220  size  and  complexity,  let  us  give 
some  numbers.  The  Datalink  (Network)  layer  specification 
consists  of  69  (19)  documents  describing  architecture,  inter¬ 
faces,  EFSM,  and  state  table  of  each  module.  The  Datalink 
layer  specification  is  accompanied  by  three  (for  Datalink 
classes  A,  B,  and  C)  Estelle  source  code  files  with  approxi¬ 
mately  1600,  8700,  and  2400  lines  of  code,  respectively. 
The  Estelle  source  code  for  the  Network  layer  has  7150 


fines  of  code,  defining  34  states  and  370  transitions  in 
seven  EFSMs. 

Due  to  its  large  size,  it  is  not  possible  to  include  the  actual 
Estelle  specifications  in  this  paper.  For  a  more  detailed 
description  of  the  semantics  of  Estelle  specification  com¬ 
ponents  (communication  channels,  interactions,  etc.),  the 
reader  may  consult  our  paper  on  Datalink  Layer  specifi¬ 
cation  [2],  or  may  visit  the  Web  site  at  http://www.cis.ude- 
l.edu/~amer/CECOM.  In  the  next  section,  we  present  an 
overview  of  the  Network  Layer  architecture  with  a  focus 
on  the  Topology  Update  and  Source  Directed  Relay  func¬ 
tions  of  the  Intranet  sublayer. 

5.1.  Intranet  layer  architecture 

Fig.  4  shows  the  interface  and  general  architecture  of  the 
Network  layer.  The  architecture  represents  the  protocol 
stack  at  a  single  station,  as  well  as  an  interface  with 
operator  “module”  which  can  interact  with  several  different 
layers  in  the  stack.  The  operator  module  abstracts  the  link 
layer’s  interactions  with  both  a  human  operator  and  a 
system  management  process.2 

Fig.  5  shows  the  internal  structure  of  the  Intranet  Layer. 
The  two  main  Intranet  Layer  functionalities,  Source  Direc¬ 
ted  Relay  (SDR)  and  Topology  Update  (TU)  exchange, 
were  encapsulated  in  separate  component  modules  of  the 
Intranet  Layer  module.  This  simplifies  the  design  of  the 


2  Note  that  the  numbers  in  Figs.  4  and  5  refer  to  interactions,  and  are 
consistent  throughout  the  figures  (e.g.  number  12  refers  to  OP-min-update- 
per  in  all  three  figures). 
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Fig.  5.  Intranet  layer  architecture. 


FSMs  that  model  the  entire  layer,  and  also  allows  for  gener¬ 
ating  test  cases  for  each  functionality  separately. 

The  SDR  module  receives  IL_Unitdata_Req  messages 
through  SNDCFSAP  interaction  point.  It  starts/stops  a  vary¬ 
ing  number  of  END_END_ACK  timers,  one  for  each  IP 
packet  that  has  been  sent  but  not  yet  acknowledged.  The 
TU  module  interacts  with  the  SDR  module  by  notifying  it  of 
any  topology  changes  that  take  place  dynamically.  The  TU 
module  communicates  with  two  timers:  Topology JUpdate 
Timer  and  Topology_Update_Re quest  Timer.  The  former  is 
started  after  a  topology  update  message  is  sent  by  the 
station.  According  to  188-220A,  a  station  is  not  allowed 
to  send  another  topology  update  message  until  the  timer 
expires.  The  latter  performs  the  same  role  for  topology 
update  request  messages. 

Both  SDR  and  TU  modules  can  send  and  receive 
messages  from  the  datalink  layer  through  their  lower_mux 
interaction  points — the  messages  from  the  two  modules  are 
multiplexed  by  the  parent  Intranet  Layer  module.  A  peer 
operator  or  management  component  is  connected  directly 
to  the  Topology  Update  module  and  can  set  parameters  that 
are  relevant  in  topology  update  mechanism.  Part  of  the 
diagram  inside  the  dash-lined  rectangular  contains  modules 
that  handle  XNP  procedures:  joining  and  leaving  the  net 
with  either  centralized  or  distributed  control,  and  parameter 
update  requests. 

5.2.  Problems  and  ambiguities  found  in  188-220  through 
formal  specification 

Primary  goals  in  developing  an  Estelle  formal  specifi¬ 
cation  are: 

1.  discover  and  document  problems  and  ambiguities  that  are 
commonly  seen  in  a  standard  written  in  natural  language; 

2.  verify  the  protocol; 

3.  simulate  the  protocol; 

4.  automate  code  implementation; 

5.  automate  test  generation  process. 


MIL-STD  188-220  project  focuses  on  goals  (1),  (3),  and  (5), 
with  simulation  studies  done  by  the  US  Army  as  reported  in 
Ref.  [19].  Although  the  formal  verification  of  188-220  is  not 
part  of  the  project,  some  of  the  errors  found  during  the 
formal  specification  can  also  be  classified  as  part  of  goal 
(2).  Achieving  goal  (4)  is  an  open  issue;  manufacturers,  who 
were  already  developing  implementations  before  the  Estelle 
specification  was  created,  now  have  an  option  to  use  the 
Estelle  specifications  for  automated  code  generation. 

In  the  process  of  developing  the  Estelle  specifications  of 
the  Data  Link  layer  and,  most  recently,  the  Intranet  Layer, 
more  than  fifty  problems  in  the  original  English  specifi¬ 
cation  have  been  documented.  Here  we  present  a  represen¬ 
tative  cross-section  of  examples  of  ambiguities  found  and 
corrected,  demonstrating  the  difficulty  of  defining  protocol 
operations  in  a  natural  language. 

Examples  range  from  ambiguities  such  as: 

•  “...a  station  shall  wait  for  some  period  of  time  bounded 
by  the  probability  of  the  remote  ack  time  expiration.” 

•  the  Intranet  Layer  allows  a  station  to  enter  Quiet  mode 
whereas  the  Data  Link  layer  refers  to  a  station  being  in 
response  mode  off.  It  was  unclear  how  these  two  terms 
differ,  if  at  all; 

to  more  serious  examples  of  correctness/completeness  such 
as: 

•  Intranet  routing  was  originally  defined  based  on  spanning 
trees  of  the  Intranet  topology.  However  the  draft  stan¬ 
dard’s  examples  did  not  comply  with  the  mathematical 
definition  of  a  spanning  tree. 

•  The  phrase  “may  report  to  the  higher  layer  protocol,  and 
may  initiate  appropriate  error  recovery  action”  was  added 
in  several  locations  when  the  datalink  layer  identified  an 
error  condition  such  as  a  lack  of  acknowledgment  after 
the  maximum  allowed  number  of  retransmissions. 

A  few  more  selected  examples  are  presented  in  Appendix  A 
to  illustrate  the  nature  of  errors  found  in  the  protocol.  All 
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Fig.  6.  Test  generation  from  extended  FSMs. 


these  problems  were  reported  back  to  the  CNR  Working 
Group  and  subsequently  removed  from  the  standard. 


6.  Test  case  generation 

Test  scripts  (test  cases)  specify  a  logical  sequence  of  test 
steps  that  are  performed  by  a  Conformance  Tester  to  indi¬ 
vidually  test  a  given  protocol  entity.  The  test  scripts  are 
input  to  the  Conformance  Tester  which  in  turn  stimulates 
an  IUT,  and  assesses  the  IUT’s  responses  to  determine  if  the 
IUT  correctly  implements  the  protocols.  Since  it  is  impos¬ 
sible  to  exhaustively  test  an  implementation  in  practice,  a 
good  set  of  test  scripts  should  at  least  check  those  events  that 
affect  state/transition,  boundary  conditions,  and  stress 
points.  The  test  scripts  themselves  should  be  structured  as 
independent  modular  components  to  facilitate  modifying 
and  adding  to  the  scripts  in  response  to  1 88-220’ s  continu¬ 
ing  evolution. 

A  number  of  techniques  have  been  proposed  to  generate 
test  sequences  from  Estelle  specifications  [42,43,61,62,74]. 
However,  full  Estelle  specifications  of  large  systems  may 
prove  to  be  too  complex  for  direct  test  case  generation.  As 
shown  in  Fig.  6,  there  are  several  ways  of  generating  test 
sequences  from  Estelle  specifications.  One  approach  would 
be  to  expand  Estelle’s  EFSMs  thereby  converting  them  to 
pure  FSMs.  This  would  be  useful  since  methods  exist  for 
generating  tests  directly  from  pure  FSMs  (e.g.  [1]).  Unfor¬ 
tunately,  converting  even  simple  EFSMs  can  result  in  the 
state  explosion  problem,  that  is,  the  converted  FSM  may 
have  so  many  states  and/or  transitions  that  either  it  takes 
too  long  to  generate  tests,  or  the  number  of  tests  generated  is 
too  large  for  practical  use. 

As  an  alternative,  UD  and  CCNY  joint  group  uses  an 
intermediate  approach,  where  an  Estelle  EFSM  is  partially 
expanded  (hence  resulting  in  some  more  states  and  tran¬ 
sitions),  but  not  expanded  completely  to  a  pure  FSM.  The 
EFSM  is  expanded  partially  just  enough  to  generate  a  set  of 
tests  that  is  feasible  and  practical  in  size.  Determining  which 


features  to  expand  in  the  general  case  is  the  difficult  aspect 
of  this  research. 

Test  Case  Generation  Research :  Conformance  test 
generation  techniques  reported  in  the  literature 
[1,7,39,46,55,62],  using  a  deterministic  finite-state  machine 
(FSM)  model  of  a  protocol  specification,  focus  on  the  opti¬ 
mization  of  the  test  sequence  length.  However,  an  IUT  may 
have  timing  constraints  imposed  by  active  timers.  If  these 
constraints  are  not  considered  during  test  sequence  gener¬ 
ation,  the  sequence  may  not  be  realizable  in  a  test  labora¬ 
tory.  As  a  result,  valid  implementations  may  incorrectly  fail 
the  conformance  tests  (or,  nonconformant  IUTs  may  incor¬ 
rectly  pass  the  tests). 

Another  problem  in  test  sequence  generation  is  due  to  the 
limited  controllability  of  an  IUT.  Typically,  the  inputs 
defined  for  the  interfaces  with  upper  layers  or  with  timers 
cannot  be  directly  applied  by  the  tester.  In  this  case,  the 
testability  of  an  IUT  may  severely  be  reduced;  in  addition, 
non-determinism  and/or  race  conditions  may  occur  during 
testing. 

In  Sections  6.1  and  6.2,  the  research  results  to  eliminate 
the  timing  constraints  and  controllability  problems  which 
appear  in  the  EFSM  model  of  the  188-220  are  outlined. 
The  current  research  is  focusing  on  the  so-called  conflicting 
timers  problem,  where  infeasible  test  sequences  may  be 
generated  unless  conflicting  conditions  based  on  timers 
are  resolved,  as  described  in  Section  6.3. 

6.1.  Timing  constraint  problem 

During  testing,  traversing  each  state  transition  of  an  IUT 
requires  a  certain  amount  of  time.  A  test  sequence  that 
traverses  too  many  self-loops  (a  self-loop  is  a  state  transition 
that  starts  and  ends  at  the  same  state)  in  a  given  state  will  not 
be  realizable  in  a  test  laboratory  if  the  time  to  traverse  the 
self-loops  exceeds  a  timer  limit  as  defined  by  another  tran¬ 
sition  originating  in  this  state.  In  this  case,  a  timeout  will 
inadvertently  trigger  forcing  the  IUT  into  a  different  state, 
and  thereby  disrupting  the  test  sequence  before  all  of  the 
self-loops  are  traversed.  If  this  unrealizable  test  sequence  is 
not  avoided  during  test  generation,  most  IUTs  will  fail  the 
test  even  when  they  meet  the  specification.  Clearly,  this  is 
not  the  goal  of  testing.  Therefore,  a  properly  generated  test 
sequence  must  take  timer  constraints  into  account. 

The  presented  methodology  [66,67]  optimizes  the  test 
sequence  length  and  cost,  under  the  constraint  that  an  IUT 
can  remain  only  a  limited  amount  of  time  in  some  states 
during  testing,  before  a  timer’s  expiration  forces  a  state 
change.  The  solution  first  augments  an  original  graph  repre¬ 
sentation  of  the  protocol  FSM  model.  Then  it  formulates  a 
Rural  Chinese  Postman  Problem  solution  [44]  to  generate  a 
minimum-length  tour.  In  the  final  test  sequence  generated, 
the  number  of  consecutive  self-loops  never  exceeds  any 
state’s  specified  limit.  In  most  cases,  this  test  sequence 
will  be  longer  than  one  without  the  constraint  since  limiting 
the  number  of  self-loop  traversals  likely  requires  additional 
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visits  to  a  state  which  otherwise  would  have  been 
unnecessary. 

The  methodology  uses  UIO  sequences  for  state  verifi¬ 
cation.  However,  the  results  presented  also  are  applicable 
to  test  generation  that  uses  distinguishing  or  characterizing 
sequences.  Earlier  results  of  this  study,  limited  to  verifica¬ 
tion  sequences  that  are  self- loops,  are  presented  in  Ref.  [66]. 
The  later  paper  [67]  generalizes  these  earlier  results  to  both 
self-loop  and  non-self-loop  verification  sequences. 

6.1.1.  Practical  motivation 

Examples  of  protocols  that  contain  many  self-loop  tran¬ 
sitions  in  their  FSM  models  include  ISDN  Q.931  for  supple¬ 
mentary  voice  services,  MIL-STD  188-220  [18]  for  Combat 
Net  Radio  communication,  and  LAPD  [68],  the  data  link 
protocol  for  the  ISDN’s  D  channel.  For  example,  in  ISDN 
Q.931  protocol  (Basic  voice  services,  for  the  user  side),  each 
state  has  an  average  of  nine  inopportune  transitions,  which 
requires  the  traversal  of  18  self-loop  transitions  during  test¬ 
ing.  A  Q.931  implementation  has  several  active  timers  that 
are  running  in  certain  states,  e.g.  timer  7304  running  in  state 
Overlap  sending ,  and  timer  7310  in  state  Outgoing  call 
proceeding.  An  EFSM  modeling  the  TU  functionality  of 
the  Intranet  Layer  has  three  active  states  in  which  one  or 
two  timers  are  running  [66]. 

It  is  not  always  possible  to  delay  the  timeout  at  a  tester’s 
convenience.  In  real  protocols,  there  may  be  timers  whose 
timeouts  are  difficult  to  set  by  the  tester,  e.g.  acknowledg¬ 
ment  timers’  timeout  values  often  are  computed  by  the 
implementation.  Moreover,  a  tester  may  want  to  test  an 
IUT’s  behavior  for  different  settings  of  the  IUT’s  internal 
timers,  to  be  able  to  test  the  IUT’s  correctness  for  various 
configurations  of  the  timers. 

In  addition  to  the  original  self-loops  of  a  specification 
model,  additional  self-loops  are  typically  created  when  gener¬ 
ated  test  sequences  use  state  verification  techniques  such  as 
unique  input/output  (UIO)  sequences  [54],  distinguishing 
sequences  [5,38],  or  characterizing  sequences  [5,38]. 


Eself  incident  on  v,.  Let  dmin_seif(Vj )  be  the  minimum  number 
of  times  any  tour  covering  all  edges  of  Evnsl  U  Eself  must 
include  vertex  v(-  £  V. 

Let  dstate_ver{Vj)  be  the  number  of  self-loop  transitions 
used  to  verify  whether  an  IUT  is  in  state  v,.  Suppose  that 
during  testing,  a  given  vertex  v,  £  V  can  tolerate  at  most 
max_self(Vj)  self-loops  executed  at  one  visit  to  vertex  v,. 
Attempting  to  remain  in  state  v,  to  execute  1  + 
max_self  (v d  self-loops  would  result  in  disruption  of  a  test 
sequence.  Testing  a  self-loop  transition  involves  traversing 
the  self-loop  transition  followed  by  applying  the  state 
verification  self-loop  sequence,  which  contains  dstate_ver(y^) 
transitions. 

Due  to  space  limitations,  we  are  unable  to  include  the 
detailed  derivation  of  dmin_seif  (v,).  In  Ref.  [66],  we  prove 
that  the  minimum  number  of  times  vertex  v,  must  be  visited 
in  a  test  sequence  is  as  follows: 


dmin_  _self(vi ) 


djvd  if  dself(Vi)  <  (djvd  *  ri,(v,)) 

I\Vi)  if  dself{Vi)  >  (, din(Vi )  *  -A, (v/)) 


where  dout{vi)  and  din(Vi )  are,  respectively,  the  out-degree 
and  the  in-degree  of  vertex  v,  in  Evnsh  and  where 


I\Vi)  =  dm(Vi)  + 


'  dsejf{yi)  -  (din(Vi)  *  ririyQ) ' 

A  2(U) 


(2) 


.  max  st 

^i(U)  =  - 


■self  (yd  d state _ver(y /) 


state  _ver(y  i) 

max_self(Vj) 


max_self(Vj) 

MVi)  =  J— 7 - 7— ■ 

_  1  '  astate_ver\vi)  _ 


(3) 

(4) 


G  (V  ,E  )  ( G '  is  obtained  from  G  by  removing  self-loop 
edges)  is  converted  to  G*(V*,E*)  by  splitting  each  vertex 
v'i  £  V1  satisfying 


dmin_self(y i)  ntux(din  (  V7/) ,  doitl  (  VJ; ) ) 


(5) 


6.1.2.  Optimizing  tests  under  timing  constraints 

Let  E seif  and  Evnsi  be  the  sets  of  self-loop  and  non- self-loop 
edges  to  be  tested,  respectively.  Let  dseif(yd,  the  number  of 
self-loops  of  vertex  v„  be  defined  as  the  number  of  edges  in 


into  the  two  vertices  v*(  1  ’ ,  v*(2>  £  V*  (Fig.  7). 

Then,  v,-  is  connected  to  vy  with  a  set  of  edges 
with  cardinality  of  dmin_self(vj)  :  E\  =  (Jv'ev'  g((v-(1),  v*(2)), 
dmin_seif(vi))-  Each  edge  in  E'\  is  assigned  infinite  capacity  (3 
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eO  eO  el  e2  e2  e2  el  0  e9  e9  e9  el  2  eO  el  e3  e2  e4 
e6  e7  e6  e6  e7  el  1  e9 el  2  el  e4  e7  e6  e7  e8  e6  e7  e5  eO 

Fig.  8.  Minimum-cost  test  sequence  without  self-loop  repetition  constraint. 

and  a  zero  cost  t//.  These  fake  edges  will  force  additional 
visits  to  Vj  in  a  minimum-cost  tour  of  G. 

We  then  use  network  flow  techniques  (similar  to  Aho  et 
al.  [1])  to  maximize  the  flow  on  graph  G  with  minimum 
cost.  This  flow  defines  a  minimum-cost  tour  of  G  under 
timing  constraints. 

Example:  Consider  the  FSM  (represented  by  the  graph 
G(V,E ))  with  self-loop  transitions  shown  in  Fig.  8.  Suppose 
that  vertices  v0,  v2,  and  v3  of  the  FSM  can  tolerate  at  most 
three,  and  v\  at  most  two  self-loop  transitions  during  each 
visit.  Let  transitions  elO  and  ell  correspond  to  timeouts. 
After  either  e  1 0  or  e  1  1  is  triggered,  the  FSM  is  brought  into 
state  v3. 

UIO  sequences  and  the  values  of  max_self,  dstate_ver  and 
dmin_seif  for  vertices  v0,  Vi,  v2,  and  v3  are  as  follows: 


eO  eO  el  e2  elO  e9  e9  e9  el  2  eO  el  e2  e2  e4  e6  e7  el  1  e9  el  2 
el  e3  e2  e4  e6  e6  e7  e5  eO  el  e4  e7  e6  e7  e5  el  e4  e8  e6  e7  e5 

Fig.  9.  Minimum-cost  test  sequence  with  self-loop  repetition  constraint. 

gered  after  the  second  consecutive  self-loop  traversal  (i.e. 
max_self{v\)  =  2).  The  IUT  will  prematurely  move  into  v3 
and  the  test  sequence  will  be  disrupted. 

To  address  the  problem  of  test  sequence  disruption  due  to 
timeouts,  the  graph  of  Fig.  8  is  converted  to  the  graph  shown 
in  Fig.  9.  Since  in  this  example  all  UIO  sequences  are  self¬ 
loops,  the  simplified  conversion  presented  in  Ref.  [66]  is 
sufficient.  The  vertices  for  which  a  premature  timeout  may 
disrupt  a  test  sequence,  which  are  Vi  and  v2,  are  split  and 
then  connected  by  dmin_self(v] )  =  3  and  dmin_self(y2)  =  4 
edges,  respectively. 

Considering  the  constrained  self-loop  problem,  the  test 
sequence  for  the  graph  of  Fig.  9  is  obtained  as 

eO,  eO,  el,e2,el0, e9,  e9,  e9 ,  e!2, e0,el,  e2,  e2,  e4 ,  e6,  el. 


Vertex 

UIO 

max_self 

d state  _ver 

dmin_self 

vo 

eO 

3 

i 

2 

Vi 

e2 

2 

i 

3 

v2 

e6,el 

3 

2 

4 

V3 

e9 

3 

1 

2 

The  Chinese  postman  method  [63]  when  applied  to  the 
graph  without  any  self-loop  repetition  constraint  results  in 
the  test  sequence 

e0,e0,el,e2,e2,  e2,  e!0,g9, e9,e9, el2,e0, el, e3,e2, e4,e6, 

el ,  e6, e6, el ,  ell,e9,el2,  el,  e4,  el , e6, el,  e8, e6, el ,  e5, eO 

(6) 

containing  34  edges.  Edges  used  for  the  purpose  of  state 
verification  appear  in  bold. 

As  can  be  seen  from  the  underlined  part  of  the  above  test 
sequence,  after  el  is  traversed,  the  IUT  should  stay  in  state 
Vi  for  a  time  that  allows  at  least  three  self-loop  traversals. 
However,  this  part  of  the  test  sequence  is  not  realizable  in  a 
test  laboratory  because  the  timeout  edge  elO  will  be  trig¬ 


ell,e9,  el2,el,e3, e2, e4,  e6, e6, el,  e5, eO, el,e4, el , e6, el, 
e5,el,e4,  eS,e6,el,  e5  (7) 

containing  40  edges. 

Although  longer  than  that  of  Fig.  8,  the  test  sequence  in 
Fig.  9  is  minimum-length  with  the  introduced  self-loop 
constraint.  During  each  visit  to  vertices  v0,  v1;  v2  and  v3, 
the  number  of  consecutive  self-loop  edges  traversed  is  less 
than  or  equal  to  the  maximum  allowed  number  of  self-loop 
traversals.  Therefore,  this  test  sequence  is  realizable  in  the 
test  laboratory. 

6.2.  Controllability  problem 

Consider  a  testing  framework  where  the  interface  7] 
between  the  IUT  and  the  (AO-layer  in  the  System  Under 
Test  (SUT)  [7]  is  not  externally  accessible  (Fig.  10).  In 
other  words,  the  inputs  from  ( N  +  l)-layer  cannot  be 
directly  applied  to  the  IUT,  nor  can  the  outputs  gener¬ 
ated  by  the  IUT  be  observed  at  (TV  +  l)-layer.  Such  an 
interface  I\  is  called  semicontrollable  if  FSM\  can  be 
utilized  to  supply  inputs  to  the  IUT.  On  the  other  hand, 
the  tester  can  apply  inputs  to  the  IUT  directly  by  using 
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SUT 


Fig.  10.  Testing  (AO-layer  IUT  with  an  (N  +  l)-layer  semicontrollable  inter¬ 
face. 

a  lower  tester,  which  exchanges  iV-PDUs  with  the  IUT 
by  using  the  (TV  —  1)-Service  Provider.  The  interface  70 
between  the  lower  tester  and  the  IUT  is  therefore 
directly  controllable . 

The  methodology  presented  in  Ref.  [23]  addresses  the 
problem  of  generating  optimal  realizable  test  sequences  in 
an  environment  with  multiple  semicontrollable  interfaces. 
The  methodology  fully  utilizes  semicontrollable  interfaces 
in  an  IUT  while  avoiding  the  race  conditions.  An  algorithm 
is  introduced  in  Ref.  [23]  to  modify  the  directed  graph 
representation  of  the  IUT  such  that  its  semicontrollable 
portions  become  directly  controllable,  where  possible.  In 
the  most  general  case,  obtaining  such  a  graph  conversion 
may  end  up  with  exponentially  large  number  of  nodes. 
However,  it  is  shown  [23]  that  special  considerations  such 
as  the  small  number  of  interfaces  interacting  with  an  IUT 
and  diagnostics  considerations  make  the  problem  size  feasi¬ 
ble  for  most  practical  cases. 

6.2.1.  Practical  motivation 

As  motivation  for  solving  the  controllability  problem,  a 
real  protocol  is  considered  where  an  SUT’s  (TV  +  l)-layer 
must  be  utilized  indirectly  to  test  certain  transitions  within 
the  (AO -layer  IUT. 

188-220  focuses  on  three  layers:  Physical,  Datalink,  and 
Network.  The  Network  layer  contains  an  Intranet  sublayer. 


An  SUT  contains  the  (AO-layer  IUT  implemented  in  the 
Datalink  layer,  and  the  Intranet  sublayer,  which  is  part  of 
the  (N  +  l)-layer,  as  shown  in  Fig.  11. 

In  the  CECOM’s  environment  used  for  testing  188- 
220  implementations,  the  upper  layers  cannot  be 
directly  controlled.  Therefore,  the  IUT’s  transitions 
that  are  triggered  by  the  inputs  coming  from  the 
Network  layer  are  not  directly  testable.  An  example 
SUT  transition  that  causes  a  controllability  problem  is 
the  transition  1 1  from  the  Class  A-Type  1  Service  Data¬ 
link  module  [18,20],  shown  in  Fig.  11.  The  input/event 
field  for  this  transition  requires  a  DL_Unitdata_Req 
from  the  ( N  +  l)-layer.  Unfortunately,  the  interface 
between  the  IUT  and  the  (N  +  l)-layer  is  not  directly 
accessible  for  generating  this  input.  Initially,  it  appears 
that  transition  1 1  is  untestable. 

To  trigger  this  transition,  which  requires  the  ( N  +  1)- 
layer  to  pass  a  DL-Unitdata.Req  down  to  the  (AO-layer, 
feedback  from  the  ( N  +  l)-layer  must  be  used.  To  force  a 
DL-Unitdata.Req  from  the  ( N  +  l)-layer,  the  tester  sends  a 
PL-Unitdata.Ind  to  the  IUT  (similar  to  the  message  a  in  Fig. 
10)  that  contains  an  intranet  layer  message  telling  the 
( N  +  l)-layer  to  relay  the  frame  to  a  different  network 
node.  The  IUT  outputs  this  message  to  the  ( N  +  l)-layer 
(see  message  b  in  Fig.  10),  and  the  (N  +  l)-layer  FSM 
responds  by  outputting  the  desired  DL-Unitdata.Req 
(message  c  in  Fig.  10).  Finally,  the  datalink  layer  generates 
the  desired  output  PL-Unitdata.Req  (corresponding  to 
message  d  in  Fig.  10)  which  can  be  observed  by  the  lower 
tester. 

In  fact,  70%  of  the  transitions  the  Class  A-Type  1  Data¬ 
link  Service  module  are  based  on  not  directly  controllable 
inputs.  Without  indirect  testing,  test  coverage  would  be 
seriously  limited — only  approximately  200  transitions  out 
of  750  would  be  testable.  However,  by  applying  the  techni¬ 
que  outlined  in  this  paper,  over  700  of  defined  transitions 
(>95%)  can  be  tested.  The  application  of  the  presented 
technique  to  188-220  is  described  in  detail  in  Ref.  [21]. 

Similar  controllability  problems  can  also  be  pointed  out 
in  testing  the  IEEE  802.2  EEC  Connection  Component 
[23,32], 


Transition  Input/Event 


Output/Action 


tl  DL-Unitdata.Req  PL-Unitdata.Req, 

Start  acknowledgment  timer 


(N+1)-layer 


(N)-layer 


Tester’s  inputs/outputs 


SUT 


Intranet 

DL- Unite 

Sublayer 

ata.Req 

Datalin 

- 4 

k  Layer 

ft - 

Fig.  11.  MIL-STD  188-220:  example  of  the  controllability  problem. 


M.A.  Fecko  et  al.  /  Computer  Communications  23  (2000)  1196-1213 


1205 


Class  1 :  Class  2:  Class  3: 


Class  4a:  Class  4b: 


i  y 

T 


Fig.  12.  Classes  of  edges  in  G'  (dashed-lined  outputs  are  optional). 


6.2.2.  Optimizing  tests  with  multiple  semicontrollable 
interfaces 

To  optimize  tests  with  multiple  semicontrollable  inter¬ 
faces,  modeling  SUT  as  a  single  FSM  was  proposed 
[22,23].  A  semicontrollable  interface  /,  is  implemented  as 
a  separate  FIFO  buffer.  During  testing,  a  buffer  may  be 
empty  or  store  an  arbitrary  sequence  of  inputs  to  the  IUT 
generated  indirectly  through  /,.  For  each  /,,  we  define  vari¬ 
able  tOj  that  has  a  distinct  value  for  each  permutation  of 
inputs  that  the  /th  buffer  can  hold.  The  proposed  model 
consists  of  graph  G  (which  represents  the  IUT’s  FSM)  and 
the  variables  oq,  a>2, ...,  coF. 

An  FSM  modeling  the  SUT  can  be  obtained  by  expanding 
G  and  oq,  cu2, ...,  coF  into  G'{V',E').  An  algorithm  for 
converting  G(V,E )  to  G'iV^E')  proceeds  as  follows  (a 
detailed  description  of  the  algorithm  along  with  its  pseudo¬ 
code  is  available  in  Refs.  [22,23]): 

Step  0 — Definitions: 

Let  Bj  denote  a  sequence  of  inputs  buffered  at  the  /th 
semicontrollable  interface.  Each  state  v'  G  V'  has  two 
components:  the  original  state  v  G  V,  and  the  current 
configuration  of  E  buffers,  i.e.  v'  =  (y,Bx,  ...,BF).  The 
algorithm  constructs  all  possible  buffer  configurations 
with  up  to  hj  inputs  buffered  at  /,. 

Step  1 — Initialize: 

r',  root  of  Gr,  as  (r,0,...,0)  (root  of  G  and  configuration 
of  empty  buffers);  E'  as  empty  set;  V'  as  { r'  j ;  Q.  queue  of 
vertices,  as  V1 . 

Step  2 — Repeat  until  Q  is  empty: 

(1)  extract  v1  =  (vstart,Bl,  ...BF)  as  first  element  from 
Q,  where  (BX,...BF)  is  current  configuration 

(2)  given  the  current  vertex  v'  =  (vstart,Bx,  ...Bp), 
perform  the  following  steps  for  each  original  outgoing 
edge  e  (V start d end)  ^  E  ■ 

o  create  new  configuration  (B\ ,  ...BF)  based  on  the  class 


of  e  (Fig.  12): 

Class  1 :  e  is  triggered  by  an  input  from  and  gener¬ 
ates  output(  s)  to  an  LT; 

Class  2:  e  is  triggered  by  an  input  from  an  LT  and 
generates  an  output  oqj  (buffered  in  Bq  to  create  a 
new  configuration)  at  Iq; 

Class  3:  e  is  triggered  by  apk  (extracted  from  Bp  to 
create  a  new  configuration)  from  Ip  and  generates 
output(s)  to  an  LT; 

Class  4:  e  is  triggered  by  an  input  apk  from  Ip  and 
generates  an  output  oqJ  at  Iq.  Apply  rules  for  Class  3 
and  Class  2  to  create  a  new  configuration, 
o  create  new  vertex  v'new  =  (vend,Bx,  ...BF)  G  V1,  and 
new  edge  e'new  =  (v',  v'new)  G  E1 
o  include  new  edges  in  E'  iff  inputs  in  (Bl,...BF)  cannot 
trigger  other  edges  outgoing  from  vstart 
o  append  to  Q  end  vertices  v'new  G  V'  of  new  edges 
included  in  E1 . 

Step  3 — Retain  only  strongly  connected  states: 

remove  from  V'  all  vertices  from  which  r  cannot  be 
reached,  and  remove  from  E'  all  edges  incident  to  such 
vertices. 

Based  on  the  practical  considerations  discussed  in  Ref. 
[23],  the  algorithm  can  be  refined  to  meet  the  following 


Table  1 

Inputs  and  outputs  for  the  edges  of  Fig.  13.  A?x  denotes  receiving  input  x 
from  A.  B\y  denotes  sending  output  y  to  B 


Edge 

Input 

Output 

Edge 

Input 

Output 

el 

LT  ?x. 

FSMx\ou 

e6 

LTlxf, 

LT\yb 

e2 

LT  1x2 

FSM21o2,  i 

el 

LTlx-i 

LT\yi 

e3 

FSM{!au 

LT\y3 

e8 

FSM{1a]2 

LT\ys 

e4 

FSM{ian 

FSMx\oh2 

e9 

LT1x9 

LTly9 

e5 

LT  ?X5 

FSM.2 !  02,2 

eW 

LT  lx io 

LT\y  io 
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SUT 


Fig.  13.  IUT  interacting  with  two  semicontrollable  interfaces. 


objective:  “ generate  a  test  sequence  that,  at  any  point  in 
time,  avoids  storing  more  than  one  input  in  only  one  of  the 
buffers  ( where  possible ).”  Satisfying  this  objective  yields  a 
linear  running  time  in  the  number  of  semicontrollable  inter¬ 
faces  and  the  number  of  edges  in  G.  If  this  objective  cannot 
be  satisfied,  the  running  time  grows  and  nondeterminism 
may  not  be  avoided  during  testing. 

Example:  Consider  the  IUT  of  Fig.  13  which  is  interacting 
with  semicontrollable  FSM\  and  FSM2  through  the  semi¬ 
controllable  interfaces  f  and  /2,  respectively.  The  IUT’s 
FSM  (represented  by  graph  G)  is  described  in  Table  1. 
Transition  el,  triggered  by  input  xj  from  the  lower  tester, 
generates  output  Oy\  to  FSM\.  In  response,  FSM ,  sends  input 
a\  \  which  triggers  transition  e3.  (In  general,  a,  y  is  the 
expected  response  to  oi  r)  Transition  el,  which  is  triggered 
by  a  lower  tester’s  input  x2,  outputs  o2,i  to  FSM2,  which 
responds  with  input  a2>1  triggering  e4.  Then  e4  outputs  o12 
to  FSMi,  which  responds  with  a12  triggering  e8.  On  the 
other  hand,  transitions  e5,  e6,  el,  e9,  and  criO  can  be  trig¬ 
gered  directly  by  the  lower  tester.  e6,  el ,  e9,  and  e  1 0  do  not 
generate  outputs  to  the  semicontrollable  interfaces.  e5 
generates  output  o22  to  FSM2,  which  does  not  send  any 
input  to  the  IUT. 

After  conversion  (Fig.  14),  each  state  of  G  is  replaced 
with  at  most  four  related  states  in  G'  corresponding  to 
the  buffer  configurations  at  a  semicontrollable  interface. 
Each  edge  e  is  annotated  as  e.x,  where  x  =  0, 1,2, 3, 
depending  on  the  input  buffered  in  the  e.x’s  start 
state,  as  shown  in  Fig.  14.  The  solid  edges  in  Fig.  14  are 
the  mandatory  edges  that  are  incident  to  nodes  that  corre- 


no  inputs  buffered  a, ,  buffered 


Fig.  14.  Graph  transformation  applied  to  the  graph  of  Fig.  13.  Mandatory 
and  optional  edges  appear  in  solid  and  dashed  lines,  respectively. 

spond  to  the  case  where  both  buffers  are  empty;  the  dashed- 
line  edges  are  the  ones  that  can  be  traversed  only  when 
either  buffer  contains  an  input.  Due  to  the  practical  diagnos¬ 
tic  considerations  [23],  we  prefer  testing  edges  when  no 
inputs  are  buffered  in  semicontrollable  interfaces.  The 
Aho  et  al.  [1]  optimization  technique  gives  the  minimum- 
length  test  sequence  for  G1  shown  in  Table  2.  Steps  with 
( — *■ )  indicate  that  an  edge  is  tested  in  this  step.  Note  that, 
for  simplicity,  the  UIO  sequences  [54]  are  not  included  in 
this  sequence. 

6.3.  Conflicting  timers  problem 

Suppose  that  a  protocol  specification  defines  a  number  of 
timers,  ci,...,ck,  such  that  a  timer  c,  may  be  started  and 
stopped  in  transitions  defined  in  states  v,  and  uh  respectively. 
Each  timer  c,  can  be  associated  with  a  boolean  variable  T, 
whose  value  is  TRUE  if  c,  is  running,  and  FALSE  if  c,  is  not 
running.  Let  <f  be  a  formula  obtained  from  variables 
Tl,...,Tk  by  using  logical  connectives  A,  V,  and  -■ . 
Suppose  that  a  specification  contains  transitions  with  condi¬ 
tions  of  a  form  “if  </>”  for  some  formula  fl.  It  is  clear  that 
there  may  exist  infeasible  paths  in  the  FSM  modeling  a 
protocol,  if  two  or  more  edges  in  a  path  have  inconsistent 


Table  2 

Minimum- length  test  sequence  for  the  IUT  of  Fig.  13 


Step 

Edge 

Input 

Output 

Step 

Edge 

Input 

Output 

—  1 

el.O 

LTlxi 

FSMJou 

8 

el. 2 

LTlx-j 

LTly, 

2 

e5.l 

LT7x5 

FSMi  \o22 

— » 9 

e8.2 

FSMPaK2 

LTly 8 

— *■  3 

e3.1 

FSMp.au 

LTly 3 

10 

el.O 

LT7Xl 

LTly, 

— ►  4 

e6.0 

LTlXf, 

LT\y6 

—  11 

e5.0 

LT7x5 

FSMi  lo2.2 

— *■  5 

el.O 

LTlx-j 

LTly 7 

—  12 

e9.0 

LF?x9 

LTly 9 

— <■  6 

el.O 

LT1x2 

FSMilou 

13 

elO.O 

LTlx  io 

LTlyw 

— ►  7 

e4.3 

FSMp.au 

FSMilou 

14 

e6.0 

LT7x6 

LTly 6 
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Table  3 

188-220  Datalink  tests:  a  single  step  corresponds  to  one  input/output 
exchange 


Test  set 

#  of  states 

#  of  transitions 

#  of  test  steps 

Class  A  Type  1  sendee 

General  behavior 

298 

799 

1732 

Precedence 

303 

401 

1316 

Multidestination 

112 

119 

145 

Class  C  Type  1  sendee 

General  behavior 

298 

799 

1732 

Precedence 

193 

357 

1314 

Multidestination 

112 

119 

145 

Class  C  Type  4  sendee 

General  behavior 

235 

925 

2803 

Outstanding  frames 

48 

172 

264 

Multidestination 

112 

119 

145 

conditions.  For  example,  for  transitions  el:  if  (T,)  then  {  } 
and  e2:  if{  ->  Tt)  then  {  },  a  path  el,  e2  is  inconsistent  unless 
el  sets  Tj  to  FALSE  (which  happens  when  timer  c,  expires  in 
transition  el).  The  problem  of  generating  tests  free  of  such 
conflicts  is  called  the  conflicting  timers  problem. 

Note  that  the  conflicting  timers  problem  is  a  special  case 
of  the  general  feasibility  problem  of  test  sequences.  There 
are  two  distinctive  features  of  the  conflicting  timers 
problem:  (1)  time  variables  are  linear,  and  (2)  time  variable 
values  implicitly  increase  as  time  elapses.  By  considering 
these  two  features,  we  expect  to  solve  this  case  more 
efficiently  than  we  could  solve  the  general  feasibility 
problem;  hence,  it  is  beneficial  to  formulate  this  problem 
separately. 

188-220  Datalink  Layer  defines  several  timers  that  can 
run  concurrently  and  affect  behavior  of  the  protocol.  For 
example,  BUSY  and  ACK  timers  may  be  running  indepen¬ 
dently  in  a  FRAME_BUFFERED  state.  If  either  timer  is 
running,  a  buffered  frame  cannot  be  transmitted.  If  ACK 
timer  expires  while  BUSY  timer  is  not  running,  a  buffered 
frame  is  retransmitted.  If,  however,  ACK  timer  expires 
while  BUSY  timer  is  running,  no  output  is  generated. 

In  the  test  cases  delivered  to  CECOM,  such  conflicts  are 
handled  by  manually  expanding  EFSMs  based  on  the 
number  of  conflicting  timers.  An  efficient  solution  to  the 
conflicting  timers  problem  that  eliminates  the  redundancies 
of  manual  state  expansion  is  expected  to  significantly 
shorten  the  test  sequences  while  providing  the  same  fault 
coverage. 

Similar  inconsistencies,  but  based  on  arbitrary  linear 
variables,  are  present  in  EFSMs  modeling  VHDL  speci¬ 
fications  [64].  Uyar  and  Duale  presented  algorithms  for 
detecting  [64]  and  removing  [65]  inconsistencies  in 
VHDL  specifications.  Current  research  in  UD  and 
CCNY  is  focused  on  adapting  these  algorithms  for  detecting 
and  removing  inconsistencies  caused  by  a  protocol’s 
conflicting  timers. 


7.  Results:  generation  and  delivery  of  test  scripts 

Using  the  initial  results,  UD  and  CCNY  collaboratively 
generated  tests  for  the  SAP  components  of  1 88-220’ s  Data 
Link  Layer  Class  A.  Class  A  stations  implement  Type  1 
(Unacknowledged  and  Coupled  Acknowledged  Datalink 
Connectionless)  Service,  with  the  original  EFSM  consisting 
of  1  state  and  15  transitions.  Based  on  the  Class  A  SAP 
functionalities,  the  original  EFSM  was  divided  into  three 
EFSMs  modeling:  (1)  general  behavior  of  the  SAP 
component  interacting  with  two  destinations;  (2)  datalink 
precedence;  and  (3)  an  IUT’s  behavior  when  interacting 
with  up  to  16  destinations.  Since  the  total  number  of 
states/transitions  that  would  be  obtained  after  full  expansion 
to  a  pure  FSM  was  infeasibly  large,  each  of  the  three  EFSMs 
was  expanded  to  a  form  closer  to  a  pure  FSM,  but  still 
containing  some  extensions. 

To  avoid  state  explosion  problem,  each  expanded  EFSM 
focused  on  certain  protocol  functionalities  while  restricting 
others.  For  example,  in  188-220,  a  sender  can  interact 
with  up  to  16  destinations,  each  of  which  may  be  free 
or  busy.  In  general  behavior  tests,  destinations  are 
allowed  transit  between  free  or  busy  mode,  but  the  sender 
is  restricted  to  communicate  with  at  most  two  of  them.  In 
multidestination  tests,  the  sender  communicates  with  up 
to  16  destinations,  which  are  forced  to  remain  in  free 
mode  at  all  times. 

Each  expanded  EFSM  was  then  used  in  automated 
test  generation.  Table  3  shows  the  sizes  of  the  expanded 
EFSMs  and  the  tests  that  were  generated  from  them. 
For  example,  the  precedence  tests  set  for  Class  A- 
Type  1  Service  was  based  on  an  expanded  EFSM  of 
303  states  and  401  transitions.  The  minimum-length  test 
sequence  generated  for  this  machine  consists  of  1316 
input/output  pairs  covering  every  transition  in  the  expanded 
EFSM  at  least  once. 

In  1997,  these  Class  A  tests  were  delivered  to  CECOM 
for  use  in  its  188-220  testing  facility.  Fig.  15  shows  a 
sample  of  the  delivered  test  scripts.  The  figure  depicts 
the  test  group  #92  from  Datalink  Class  A-Type  1 
service  tests.  Each  test  group  is  a  subsequence  of  a 
full  test  sequence  that  starts  and  ends  in  the  initial 
state.  In  the  first  step,  the  technique  of  utilizing  semi- 
controllable  interfaces  presented  in  Section  6.2  is  used. 
The  lower  tester  sends  a  packet  with  three  destination 
addresses:  IUT_addr,  des_addr_l,  and  des_addr_2.  The 
setting  Relay  =  Yes  in  the  INTRANET  clause  tells  the 
first  addressee,  i.e.  the  IUT,  to  relay  the  packet  to  the 
two  remaining  addressees.  As  a  result,  the  IUT  sends  a 
packet  with  its  address  as  a  source,  and  des_addr_l  and 
des_addr_2  as  destinations,  as  if  it  were  originated  by 
the  IUT’s  Intranet  Layer.  In  the  second  and  third  steps, 
the  IUT’s  packet  sent  in  the  first  step  is  acknowledged 
by  des_addrJ2  and  des_addr_l,  respectively.  Each  test 
step  is  further  annotated  with  the  test  description,  the 
number  of  the  corresponding  Estelle  transition(s),  and 
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II  Test  Group  #92 

II . 

TESTGROUP=92; 

LAYER=Datalink; 

II  Test  1 

STIMULUS=send;  II  PL-Unitdata.lnd 
TIME=tong; 

//  DL1 

INTRANET={ 

Type=IP; 

LowDelay=Yes; 

HghThroughput=No; 

HlghReliability=No; 

Precedence=1 ;  //  PRIORITY 
0rgAddr=des_addr_1 7; 
DestRelay={ 

AddrdUTaddr; 

Distance=1 ; 

Des=No; 

Relay=Yes; 

Ack=No; 


DestRelay={ 
Addr=des_addr_1 ; 
Distanced; 
Des=Yes; 
Relay=No; 
Ack=No; 

}; 

DestRelay={ 

Addr=des_addr_2; 

Distanced; 

Des=Yes; 

Relay=No; 

Ack=No; 


DATALINK={ 

CtrlField={ 

SendSeq=l ; 

RecSeq=1 ; 
ControlSpare=1 ; 
DLPrec=1 ;  //  PRIORITY 
IDNum=1; 

PDU=ui_0; 

}; 

Conmand=Yes; 
SrcAddr=des_addr  J 7 ; 
DestAddr=IUT_addr; 


RESULTS=receive;//  PL-Unhdata.Req 
TIME=normal; 

II DL1 

DATALINK={ 

CtrlReld={ 

SendSeq=1 ; 

RecSeq=1 ; 

ControlSpare=1 ; 

DLPrec=1 ;  //  PRIORITY 
IDNum=1; 

PDU=ui  _1 ; 

}; 

Command=Yes; 

SrcAddr=IUT_addr; 
y  DestAddr=des_addr_1  ;des_addr_2; 


TESTDESCRIPTION={ 

Intranet  layer  passes  down  a  multidestinati on  packet 
which  is  queued  by  datalink  layer.  Packet  requires 
a  coupled  ack.  There  are  no  outstanding  frames. 

No  outstanding  frame.  Queued  frame  transmitted  1o  multiple 
destinations.  Frame  requires  a  coupled  ack.  Ack  timer 
started. 

}; 

//  ESTELLE  TYPE1  SAP_3,4,TYPE1  SAP  _1 8 

//  SECTION(S)  5.3.16_5.3.6.1.1_C4.3,5.3.4.2.2.2.1_5.3.6.1.1 

II  Tesl  2 

STIMULUS=send;  II  PL-Unitdata.lnd 
TIME=normal; 

//  DL1 

DATAL1NK={ 

ClrlField=( 

SendSeq=1 ; 

RecSeq=1 ; 

ControlSpare=1 ; 

DLPrec=2;// ROUTINE 
IDNum=1; 

PDU=urr_0; 

}; 

Command=No; 

SrcAddr=des_addr_2; 

Desl  Addr= I  UT_addr ; 

}; 

RESULTS=noop;  II  none 
TESTDESCRIPTION={ 

Second  destination  acks  a  multidestination  packet. 

Rrst  has  not  acked  yet. 

); 

//ESTELLE  TYPE1SAP_12 
II  SECTION(S)  5.3.7. 1 ,5.5_5.3.6.1 ,6_C4.3 

II  Test  3 

STIMULUS=send;  II  PL-Unitdata.lnd 
TIME=normal; 

//  DL1 

DATALINK={ 

ClrlField={ 

SendSeq=1 ; 

RecSeq=1 ; 

ControiSpare=1 ; 

DLPrec=2;// ROUTINE 
IDNum=1; 

PDU=urr_0; 

}; 

Command=No; 

SrcAddr=des_addr_1 ; 

Dest  Addr= I  UT_addr ; 

}; 

RESULTS=noop;  //  none 
TESTD  ESCRI  PTION={ 

Rrst  destination  acks  a  packet.  Ack  timer  is  stopped. 

No  Irame  queued  lor  transmission. 

}; 

//ESTELLE  TYPE1SAPJ2 
//  SECTION(S)  5.3.7.1.5.5_5.3.6.1.6_C4.3 


Fig.  15.  A  sample  of  test  scripts  delivered  to  CECOM. 


the  appropriate  section(s)  from  the  188-220  official 
document. 

Since  the  summer  of  1997,  the  work  on  test  generation 
expanded  to  include  Class  C.  Class  C  also  allows  Type  1 
Service  as  in  Class  A,  but  it  additionally  defines  Type  4 


(Decoupled  Acknowledged  Connectionless)  Service.  As  in 
the  case  of  Class  A,  three  EFSMs  were  used  to  generate 
three  sets  of  tests  for  each  Class  C  service.  The  sizes  of 
the  EFSMs  and  the  corresponding  minimum-length  tests 
are  shown  in  Table  3.  For  example,  the  general  behavior 
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Fig.  16.  Estelle  as  part  of  synergistic  efforts  to  develop  C41  systems. 


which  resulted  in  the  development  of  new  testing  methodol¬ 
ogies.  By  applying  these  new  results,  the  conformance  tests 
for  188-220  were  generated  while  the  protocol  was  still 
evolving.  Performing  initial  conformance  tests  on  proto¬ 
types  uncovered  several  interoperability  errors  very  early 
in  the  development  process. 

Following  this  success  of  the  188-220  development,  the 
synergistic  efforts  to  develop  C4/  systems  with  the  help  of 
formal  methods  serves  as  a  model  for  DoD  standards 
process  and  development  for  the  future  [19]. 

8.2.  Advantages  of  formal  methods  in  eliminating  protocol 
errors 


tests  set  for  Class  C  Type  4  Service  was  based  on  an  EFSM 
of  235  states  and  925  transitions.  The  minimum-length  test 
sequence  generated  for  this  machine  consists  of  2803  input/ 
output  pairs.  These  tests  were  delivered  to  CECOM  in  1998. 

At  the  time  this  paper  was  written,  the  implementations 
from  three  major  manufacturers  were  being  tested  at 
CECOM.  The  tests  generated  by  the  UD  and  CCNY  team 
have  uncovered  several  implementation  errors,  including 
lack  of  mandatory  capabilities  in  Datalink  layer,  and 
problems  with  multi-hop  Intranet  Relaying. 


8.  Conclusions:  improvements  to  protocol  development 
process 

8.1.  Integration  of  Estelle  into  system  development 

Traditional  sequential  process  of  system  development  is 
known  to  be  inefficient  since  it  allows  unnecessary  duplica¬ 
tion  and  does  not  facilitate  tracking  of  rapidly  changing 
technology.  With  188-220  as  a  critical  component,  a  syner¬ 
gistic  framework  for  C4/  (Command,  Control,  Communica¬ 
tions,  Computers,  and  Intelligence)  systems  development 
has  been  established  [19]  (Fig.  16).  It  combined  several 
parallel  activities:  developing  protocol  standards  and  speci¬ 
fications,  formally  specifying  protocols  in  Estelle,  building 
conformance  tester  hardware  and  software,  “field  testing”, 
modeling  and  simulation,  as  well  as  resolving  and  docu¬ 
menting  the  solutions  to  standards-related  technical  issues 
by  the  Joint  CNR  Working  Group.  (WG  participants  include 
representatives  from  DoD  services/agencies,  industry,  and 
academia.) 

Using  formal  methods  as  part  of  this  process  helped 
create  a  high  quality  protocol  standard,  which  is  robust, 
efficient,  and  relatively  error-free.  Also,  due  to  the  struc¬ 
tured  nature  of  Estelle,  the  specification  process  progressed 
at  an  accelerated  pace  compared  to  the  English  specifica¬ 
tions  (188-220  was  completed  on  time,  setting  a  rare  exam¬ 
ple  in  protocol  standards  arena). 

Since  it  is  relatively  easier  to  extract  modeling  informa¬ 
tion  from  a  formal  specification,  the  researchers  at  UD  and 
CCNY  were  able  to  solve  a  number  of  theoretical  problems, 


The  difficulties  of  describing  protocol  operations  with 
clarity,  precision,  and  consistency  by  using  a  natural 
language  are  illustrated  by  the  examples  in  Section  5.2.  In 
addition  to  the  vagueness  introduced  by  a  natural  language 
description,  ambiguities  and  contradictions  are  difficult  to 
detect  when  related  protocol  functionalities  are  defined  in 
different  document  sections  separated  by  pages  of  unrelated 
text.  Such  problems  are  eliminated  in  a  formal  Estelle  speci¬ 
fication.  All  actions  in  a  particular  context  are  defined  in  one 
place  within  the  Estelle  specification.  The  specifications 
make  the  conditions  for  state  transitions  explicit  through 
Estelle  constructs.  Indeed,  the  very  process  of  creating 
these  constructs  enables  formal  specifiers  to  detect  some 
of  these  types  of  ambiguities  which  are  difficult  to  see  in 
normal  reading. 

8.3.  Observations  on  applicability  of  formal  methods 

As  concluding  remarks  for  this  paper,  we  report  the 
following  observations  based  on  our  experience  during  the 
formal  specification  and  test  generation  for  188-200. 

To  develop  an  Estelle  formal  specification  of  a  protocol, 
we  must  not  only  define  its  architecture  and  interface 
components  (e.g.  as  in  Figs.  4  and  5  for  188-220),  but  we 
must  also  carefully  specify  the  behavior  of  each  module  of 
these  components.  This  definition,  achieved  through  the 
creation  of  EFSMs,  is  the  most  difficult  and  time-consuming 
step  of  creating  a  formal  specification.  A  syntax-directed 
editor  improves  the  readability  for  testers  who  are  not 
FDT-trained;  it  also  is  useful  in  writing  non-trivial  specifi¬ 
cations.  Moreover,  the  modeling  and  specification 
languages,  such  as  SDL  [25,26]  and  UML  [50],  enjoy  wide¬ 
spread  industrial  popularity,  partially  due  to  their  standard 
graphical  representation.  Therefore,  it  will  be  a  natural 
extension  for  Estelle  to  include  a  graphical  editor  (such  an 
editor  has  recently  been  created  [56]).  Once  all  states  and 
transitions  of  a  protocol  (including  inputs  and  outputs)  are 
finalized,  the  writing  of  the  Estelle  code  itself  is  fast  and 
straightforward. 

Since  188-220  is  a  multilayer,  multifunction  protocol  of  a 
considerable  size  and  complexity,  manual  generation  of 
conformance  test  sequences  would  be  both  inefficient  and 
ineffective.  As  seen  from  Table  3,  the  tests  already  delivered 
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to  CECOM  contain  approximately  10,000  test  steps.  It  is 
clear  that  manually  generating  test  sets  of  this  size  from  the 
protocol  textual  description  is  not  a  trivial  task. 

A  number  of  conformance  test  generation  techniques 
have  been  proposed  [1,7,9,46,53,55,59,62],  each  of  which 
is  expected  to  give  better  results  for  a  certain  class  of  proto¬ 
col  specifications  depending  on  the  nature  and  size  of  the 
protocol.  The  experience  obtained  in  generating  tests  for 
188-220  suggests  that  to  successfully  test  today’s  complex 
protocols  by  using  formal  methods,  an  ideal  test  generation 
tool  should  support  multiple  test  generation  techniques  [41]. 
They  can  range  from  Postman  tours  [1]  or  fault-oriented 
tests  [70,72]  for  mid-size  protocols  (i.e.  the  number  of  states 
in  the  order  of  thousands),  to  guided  random  walk 
approaches  [39,73]  for  large  protocols  (i.e.  the  number  of 
states  larger  than  tens  of  thousands). 

State  explosion  problem  has  been  a  major  issue  for 
generating  FSM  models  out  of  EFSM  representations  of 
protocols  [15,52,71,72].  One  common  procedure  for 
converting  EFSMs  into  FSMs  simultaneously  performs 
reachability  analysis  and  online  minimization  [15,40]; 
this  conversion  is  based  on  combining  equivalent  states 
[54]  by  using  the  notion  of  bisimulation  equivalence 
[47].  Another  approach  proposes  the  elimination  of 
inconsistencies  in  EFSM  models  [64,65].  Efficient  algo¬ 
rithms  such  as  these  should  be  implemented  in  any  test 
generation  tool  using  FSM  models.  If  the  final  FSM 
model  is  not  confined  to  a  manageable  size,  the  test 
sequences  generated  from  it  will  be  infeasibly  long 
regardless  of  the  test  generation  method. 

Finally,  a  test  house  may  require  its  own  proprietary 
format  for  the  executable  tests.  Although  TTCN  is  accepted 
as  input  by  many  test  tools,  a  proprietary  test  format  may  be 
preferable  for  a  given  protocol,  if  this  format  is  more  read¬ 
able  by  the  domain  experts,  or  is  simpler  to  parse  by  soft¬ 
ware  tools.  The  output  of  a  test  generation  tool  should  easily 
be  custom-tailored  for  a  particular  format,  possibly  by  using 
simple  application  generators. 
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Appendix  A 

A.  1.  Examples  of  errors  found  in  188-220 

•  The  standard  indicates  in  Section3  5. 3. 8.1. 2. e  that  k  is  a 

link  parameter  defined  to  be  the  “maximum  number  of 

3  Section  numbers  refer  to  the  MIL-STD  188-220  official  document. 


outstanding  I  PDU’s.”  But  in  Section  5. 3. 7. 2. 5.1,  we  see 
that  the  “...RR  command  is  sent  to  a  destination  station 
when  the  k  value  at  the  originating  station  reaches  half  of 
the  k  value  for  that  connection.”  In  Section  5. 3. 7. 2. 5.1, 
the  first  reference  to  k  indicates  that  A:  is  a  variable  which 
is  being  used  as  a  counter,  while  the  second  reference  to  k 
has  k  being  a  static  parameter  for  that  connection.  In  this 
case,  the  standard  was  changed  such  that  the  variable 
counting  the  number  of  outstanding  I  PDUs  was  not 
referred  to  as  k. 

•  Section  5. 3. 7. 2. 5.4. 2  states  that  “When  an  I  PDU  has 
been  received  and  not  more  than  one  frame  is  missing, 
the  station  may  retain  the  information  field  of  the  out-of¬ 
sequence  I  PDUs  and  send  a  SREJ  PDU  for  the  missing  I 
PDU.”  This  sentence  implies  at  most  one  missing  I  PDU 
when  sending  an  SREJ  PDU.  The  next  sentence, 
however,  states  “A  station  may  transmit  one  or  more 
SREJ  PDUs,  each  containing  a  different  N(R)  with  P- 
bit  set  to  0.”  This  sentence  implies  that  there  can  be 
more  than  one  missing  I  PDU  (otherwise  why  send  multi¬ 
ple  SREJ  PDUs  with  different  N(R)?).  At  the  13  March 
meeting,  the  WG  changed  this  section  to  indicate  that  an 
SREJ  PDU  may  be  sent  when  at  least  one,  rather  than  “no 
more  than  one,”  I  PDU  is  detected  as  missing. 

•  In  Section  5. 3. 6. 1.9,  the  standard  indicates  that  “the 
URNR  response  PDU  shall  be  used  to  reply  to  a  UI 
command  PDU  with  the  P-bit  set  to  1,  if  the  UI  command 
cannot  be  processed  due  to  a  busy  condition.”  Since  the 
UI  cannot  be  processed,  this  indicates  the  URNR  does  not 
acknowledge  the  UI.  However,  Section  5. 3. 5. 2. 3. 2  states 
that  “the  F-bit  set  to  1  shall  be  used  to  acknowledge  the 
receipt  of  a  command  PDU  with  the  P-bit  set  to  1.”  And 
Section  5. 3. 7. 1.5. 4  indicates  that  “a  URNR  response 
PDU,  with  F-bit  set  to  1,  may  be  sent  by  the  remote 
station  to  advise  the  originator  of  the  associated  UI 
command  PDU  that  it  is  experiencing  a  busy  condition 
and  is  unable  to  accept  UI  PDUs.”  The  combination  of 
the  last  two  excerpts  implies  that  URNR  response  PDUs 
do  acknowledge  the  corresponding  UI  PDU.  A  correction 
of  this  contradiction  was  approved  at  the  13  March  96 
WG  meeting.  Section  5. 3. 5. 2. 3. 2  was  changed  to  read 
“the  F-bit  set  to  1  shall  be  used  to  respond  to  the  receipt 
of  a  command  PDU  with  P-bit  set  to  1.”  No  acknowl¬ 
edgment  is  implied. 
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